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ABSTRACT 

A qualitative evaluation related to the creation of a 
Ragional High School of Excellence (RHSE) in upstate New York is 
described. The focus is on a summer pile- study of the RHSE conducted 
while the continuation of the project was jeopardized by proposed 
budget cuts. The stages of development, problems involved in passing 
the proposal through the bureauc** and issues raised about 
designing an evaluation for c">^ • ,»o may not implement the 
evaluation are discussed. Th .:*«%j.\jition described was 
requested/commissioned by the Albany-Schenectady Schoharie Bureau of 
Occupational and Educational Services located in Albany (New York) . 
The RHSE, a pilot project for gifted and talented students in all 
academic subjects, is expected to open over the next 5 years. 
Evaluation design stages addressed conceptualization, rationale for 
choice of methods, training issues, monitoring the processes involved 
with clients, and post-implementation strategies for survival. The 
evaluation plan had a strong documentary focus involving interviews, 
documents, life histories, teacher journals, student journals, and 
surveys (including parent involvement surveys) . Ethnographic 
techniques and the use of numerous methods were central to the 
evaluation design's organization. The evaluation design encompassed 
resource inputs, processes or activities, and outcomes as compared to 
objectives. A detailed outline of action components of the 
qualitative evaluation plan of the school before, during, and after 
the program was developed; and various evaluation strategies were 
pilot tested. The aforementioned budget cuts resulted in a 
postponement of the school project. The life history in teaching 
survey, teacher evaluation survey, and parents survey instruments are 
included. (TJH) 
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I NTRODUCTION AND PURPOSES 
This paper reports the stages of development, problems 
involved in passing the proposal through the bureaucracy, and 
issues raised about designing an evaluation for clients who may 
not implement the evaluation* The evaluation was requested and 
commissioned by the Albany-Schenectady Schoharie BOCES located in 
Albany, New York, for the proposed Regional High School of 
Excellence, (RHSE). The RHSE is a pilot project for gifted and 
talented students in all academic subjects* The Honorable Mario 
M. Cuomo, Governor of New York has requested and endorsed this 
proposal and it is expected that the RHSE will begin in the next 
five years* 

As an evaluator who designed this evaluation plan, I became 
interested in the meaning of the possible and real problems 
confronting me as evaluator even though I would be part of the 
evaluation process long before the RHSE became a reality. In 
this paper I analyze the following stages in development of the 
evaluation: 

Stage one: Conceptualization, Rationale for choice of 

Methods, Training issues 
Stage Two: Monitoring the processes involved with the clients 

Initially identified as part of the project 
Stage Three: Post Implementation Strategies for survival* 

In the midst of stage Two, a major change was realized* The 
existence of the project itself was jeopardized by budget cut 
threats. 
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The evaluation plan had a strong documentary focus involving 
interviews, documents, life histories, teachers journals, student 
journals, and some video taped sequences of instruction with 
analysis. The evaluator had to design strategies to save the 
plan from extinction. The two main questions guiding the plan 
through a summer pilot of RHSE had to be addressed: 

1) What did we do in the pilot study? 

2) How well did we do it? 

The paper concludes with a description cf how the evaluator 
managed to set up and implement the summer pilot study of what 
the main evaluation might look like if the RHSE were implemented 
in the near future. The bulk of the analysis will address the 
budget cut threats which are at this writing still pending. 

THE PROPOSED REGIONAL HIGH SCHOOL OF EXCELLENCE 

(RUSE) 

The RHSE proposal grew out of the nsed to develop a model 
program to show that public schools can and should strive for 
excellence in all subjects. The high school to be located in 
upstate New York is targeted for high school students who are 
identified as gifted and talented in any number of subjects based 
on multiple measures of assessment. Early in the proposal 
generation stage, the Albany-Schsnectady-Schoharie BOCES (Bureau 
of Occupational and Educational Services) the main delivery 
system for special education, called me to work with them in 
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developing an evaluation plan from the very beginning of the 
pro j ect . 

In order to develop the evaluation plan for the RHSE, it is 
necessary to describe the choice of evaluation methodology and 
the strategies to be used in the evaluation plan. A brief 
historical overview is in order regarding choice of procedure. 
In the past, three major approaches have been employed by 
evaluators. These approaches, associated with three generation 
of evaluators, usually were based on formulas and were summative 
in nature. Currently, a fourth generation redefines evaluation 
as a socio-political process that is diagnostic, change-oriented, 
and educative for all parties involved (See Guba and Lincoln, 
1985). It is usually formative in nature. 

Guba and Lincoln, (1985), have outlined the distinguishing 
characteristics and techniques of each generation of evaluators. 

(1) THE FIRST GENERATION 

This could be called the technical generation. Major tasks 
involved stands dized measures of intelligence, aptitude, 
and achievement. 

(2) THE SECOND GENERATION 

This group has been called the descriptive generation. The 
evaluator described discrepancies between performance and 
objectives. This was the objective - focused evaluation, 
which expanded evaluation to include programs and students. 
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(3) THE THIRD GENERATION 

This group of evaluators has been called the judgmental 
generation. Evaluators used technical measurement and 
description but added standards against which program 
performance could be judged. The standards were used to 
reach a judgement which might be either formative or 
summative. This group of evaluators focused on organizers 
other than objectives, including decision models, goal free 
models, social experimentation models and cost effectiveness 
models. Stake (1973) pointed out that this approach added 
the judgmental countenance to the existing descriptive 
countenance. 



(4) THE FOURTH GENERATION 

This generation can be called the negotiative generation. 
The evaluator is not the judge, but rather the person who 
sets the stage for the judgmental process between 
respondents and evaluators. The evaluator is a teacher and 
a learner, conducting a responsive evaluation. The 
evaluator collects data pertinent to priority claims, 
concerns, and issues but does not draw conclusions. Rather, 
the evaluator prepares a report for negotiation which is 
presented to the stakeholders for their use. The evaluator 
role becomes one of mediator in the negotiation process. 
The major difference in the fourth generation approach from 

the previous three generations, is notable. First, the evaluator 
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is no longer seen as a technical expert who discovers the "true" 
state of affairs, and vrho possesses objectivity to make the 
"correct" recommendations. The evaluator is NOT THE JUDGE. 
Success is assessed in terms of: 

(a) the stakeholders increased understanding of the 
enterprise, 

(b) the utility of the proposed agenda for negotiation, 

(c) the quality of information provided and, 

(d) the expertise displayed in guiding the stakeholding 
groups through the negotiation process (Guba and 
Lincoln, 1985, p. 141) 

This fourth generation approach has its foundations in the 
qualitative approach to evaluation. This approach looks at human 
activities, behaviors, and beliefs, over time in a given social 
setting in order to understand the many realities occurring in 
that setting. The qualitative evaluator understands that in 
dealing with constructed realities, those realities can and often 
do change when brought into a position of examination. Th6 
qualitative evaluator conducts in a way a "co-evaluation," with 
the stakeholders who dictate the nature of the program. The 
tools of the qualitative evaluation are well suited to this 
posture of diagnosio, change, and mutual education. 

MAJOR TECHNIQUES OF QUALITATIVE EVALUATION 
Ethnographic research techniques have been used widely over 
the past decade in the areas of policy analysis, and educational 
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research and evaluation. Dissatisfaction with traditional 
quantitative designs, (Eisner, 1979; Filstead, 1979) had 
contributed to the awakened interest in ethnographic techniques 
which are associated with qualitative inquiry. The commonly used 
systems-analysis approach (Rivlin, 1971) , which developed in 
economics and industry, has dominated the field of educational 
evaluation. This approach focuses on measurement of variables 
which are scaled and quantified easily. In many cases, this 
approach has been inappropriate to the phenomena under 
investigation and has produced data of uneven and questionable 
validity, (Guttentag, 1977; House, 1979; LeCompte, 1970). 
Consequently, researchers and evaluators gravitated to 
ethnographic research strategies because of their higher internal 
validity (Denzin, 1978; Erickson, 1977; Raichardt and Cook, 
1979) . Literature encouraging ethnography in educational 
evaluation is prolific (Patton, 1980; Guba and Lincoln, 1981; 
Lincoln and Guba, 1985; Guba and Lincoln, 1985; Fetterman, 1984). 
The next portion of this paper will describe ethnographic 
techniques for conducting a qualitative evaluation. 



CHARACTERISTICS OF ETHNOGRAPHIES 



Ethnographies are analytic descriptions or reconstructions 
of intact cultural scenes and groups (Spradley and McCurdy, 
1972). Ethnographic work is: 

(a) holistic rather than fragmented, 
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(b) focused on the complexities of a social setting, 

(c) multimodal, using a variety of techniques, 

(d) firsthand, that is directly related to actual beliefs, 
and behaviors of participants and 

(e) phenomenological, that is the representation of the 
world view of the participants, (Janesick, 1982, 
1989) . 

Three kinds of data provided by ethnographic techniques are 
useful in assessing the impact of an intervention program or 
curricular innovation. (See LeCompte, and Goetz, 1984). The 
three kinds of data include the following: 



(1) BASELIN.? DATA: 

Information about the research setting, participants; 
social, psychological cultural, demographic and physical 
features about the social context; institutional framework 
and its relationship with other institutions should be 
examined for countervailing influences impinging "upon change 
and stability (See Apple and King, 1977; Sharp and Green, 
1975) . 

(2) PROCESS DATA: 

Information about what happened in the course of the 
curricular innovation. 

* 

(3) VALUES DATA: 
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Information about the values of the participants, program 
administrators, and policymakers who financed the program: 
what values the program supports, what is neglected. 
(Krathwohl, 1980). 

These three types of data provide evaluators and 
stakeholders with flexibility, an open and holistic design, the 
opportunity to see what actually takes place in a program and the 
option for improving or changing the status quo. 

MULTIPLE METHODS FOR QUALITATIVE EVALUATION 

There are a wide variety of methods available to the 
qualitative evaluator. (Pelto and Pelto, 1978; and Goetz and 
LeCompte, 1984) , have written thorough textbooks detailing these 
methods. For the purposes of this paper, the author will briefly 
describe these methods. The major techniques in 
qualitative/ethnogr"aphic~work include: (a) " participant ~ 
observation, (b) key informant interviews, (c) life histories, 
(d) document analysis, and (e) surveys. 

(a) Participant Observation 

Participant Observers stay in the social setting over 
time, and by taking field notes, describe the 
interactions and activities of the key participants in 
the setting. The role of participant observer can be 
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seen as a continuum where depending on circumstances, 
the participant observer may be either more participant 
or more observer or some combination of both. 

(b) Key Informant Interviews 

Key informants are the main participants who possess 
special knowledge, skills, and status in the 
investigation. Key informants provide insights and 
information that the evaluator may not otherwise have 
access to. These are people with firsthand experience 
in the social setting and may be interviewed any number 
of times in order to complete whatever inquiry is 
underway. (See Spradley, 1979, for a detailed 
description of Ethnographic interviewing.) 

(c) Life Histories 

Anthropologists have used life history interviewing or 
written histories eis a means to formulate questions or 
make inferences about a given social group or setting. 

(d) Document Analysis 

Any documents such as memos, plans, enrollment records, 
student or teacher made products, correspondence and 
the like may provide valuable process, baseline, and 
values data. Any written or pictorial data like 
photographs may also be used for analysis. 
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(e) Surveys 

Survey instruments may take many f orms such as 
structured interviews or questionnaires, participant- 
construct devices where researchers ask participants to 
sort and classify information, and projective devices. 
Projective devices may include photos, drawings or 
games designed to elicit people's opinions or 
reactions. (See LeCompte and Goetz, 1934). Stream of 
behavior chronicles may also be considered as a survey. 
These are usually videotaped or recorded on audio tape 
or handwritten on the spot, Evaluators may sample 
across events, settings and participants to study 
themes and questions. 
These major techniques can be used in program evaluation in 
a meaningful way to study innovations. For this reason a 
qualitative evaluation plan was chosen for the proposed Regional 
High School of Excellence (RHSE) . My first challenge was to 
check the existing literature on evaluation of gifted and 
talented programs and find a suitable framework for adjusting to 
our specific needs. 

A FRAMEWORK FOR THE EVALUATING THE RHSE MODEL PROGRAM 
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Picturing the program often helps in understanding 
evaluation needs. This framework (FIGURE A) has been designed by 
Sylvia Rimm (1982) for the Evaluating and Monitoring of a Gifted 
Program. Stakeholders decided that it had great value for the 
RHSE and was proposed as the framework for the evaluation plan. 



FIGURE A: FRAMEWORK FOR THE EVALUATION 
EVALUATION AND MONITORING OF A GIFTED PROGRAM 

(RIMM, 1977) 



DECISION MAKERS 



Step 1- 

INPUT (RESOURCES 
Personnel 



Step 2- 

PROCESS (ACTIVITIES) 
Teaching Techniques 
and Organization 
Books and Materials Teacher In-Service 



Equipment 



Enrichment Activities 
Parental Involvement 



Step 3- 

OUTCOME (OBJECTIVES) 

General Cognitive 

Achievement 

Specific Skills 

Achievement 

Student Attitudes 

Student Behaviors 

Parent 

Community 

Attitudes 



Step 4 « Evaluation 
This framework is almost self-explanatory. The major 
components are input, process, and outcome. Each component can 
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be evaluated using ethnographic techniques to present a 
comprehensive picture of the "\mpact of a program. This framework 
may be adapted to the individual needs of any given program at 
the final evaluation stage. To begin, decision makers need to 
keep accurate and thorough records of all planning activities as 
well as each step of this framework. Given and understanding of 
the qualitative evaluation techniques described and the concept 
of fourth generation evaluation, the remainder of this paper will 
deal with the action steps proposed before, during and after 
program implementation. Each action step will be related to the 
appropriate component of Rimm's framework. 

ACTION COMPONENTS OF THE QUALITATIVE EVALUATION PLAN OF THE RHSE, 
BEFORE DURING, AND AFTER PROGRAM IMPLEMENTATION 
I. Before Implementation 

A. Develop criteria for staff selection (Input) 

B. Develop application form for staff (Input) 

C. Require staff members to complete a written life 
history as part of the application. " Emphasis should 
include the applicants interest in teaching, particular 
interest in the RHSE, and why the person chose teaching 
as r career. See Protocol A: Life History (Input) 

D. After all books and materials are selected, rate these 
books and materials. See Protocol B: A Guide for 
Evaluating Materials for Gifted Children (Breiter, 
1982) 

E. Identify necessary equipment and facilities. (Input) 
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F. Develop criteria for student selection. Include 
alternative criteria and identification procedures for 
minority students and economically disadvantaged 
students . (Input) 

G. Document and record all efforts to identify and recruit 
minority staff, and students. (Input) 

II. During Implementation 

A. Implement teacher self appraisal techniques (1) journal 
writing and (2) evaluation of in-service training 
survey. See Protocols C and D. (Process) 

3. Implement journal writing for teaching staff to 
include: 

(1) description of activities 

(2) preparation for activities 

(3) number of participants 

(4) perceived effectiveness 

(5) modifications for the future 

(6) data collected. 

As Rimm (1982) points out, a personal log kept by all 
staff members provides formidable documentation of what occurred 
and forces participants to evaluate themselves. (Process) 
C. Develop a questionnaire for parents' attitudes 

following their child's involvement in the RHSE. 
(Process and Outcome) See Protocol E: Parents Survey 
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D, After enrichment activities are planned, develop short 
survey evaluation form aa reeded to determine its value 
to participants. (Process) See Protocol F 
III. AFTER IMPLEMENTATION 

A. The director should collect student assessment data for 
presentation in the final report. (Outcome) 

B. The Director should collect teacher self-assessment 
data for presentation in the final report. (Outcome) 

C. The Director should compile all pertinent data for a 
final report. 

STAGE TWO: PILOT TEST OF VARIOUS EVALUATION STRATEGIES 
Following the first stage of conceptualization, the 
evaluator met regularly with the Director of the Summer Program 
to select the appropriate techniques for evaluating the small 
scale summer gifted and talented program. It was decideu that 
the following techniques would be employed: 
~ a) student logs 

b) teacher journals 

c) life histories 

» d) evaluation survey 

e) parents survey (see appendices) 

Each week, the entire staff would meet to discuss all 
aspects of the program and it was decided that the Director would 
keep complete records of these meetings and be responsible for 
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including information from the meetings in a final evaluation 
report* 

The summer program was successful and participants kept 
journals faithfully after a training session on journal writing* 
The next phase was projected to be the actual start up of the 
RHSE itself* 

STAGE THREE: BUDGET CUTS AND THE POSTPONEMENT OF 
THE RHSE AND LESSONS LEARNED FROM THIS. 

Unfortunately, the New York State Budget for programs of 
this nature was substantially cut which resulted in postponement 
of the RHSE plan. Key participants still voiced their full 
support of the evaluation plan for the project itself, based on 
the summer experience. In this, the second year of limbo 
participants remain hopeful that in the near future the project 
will be implemented for various socio-political and economic 
reasons. In any event, as the evaluator, I am still in contact 
with the participants to keep updated on the situation and would 
of course like to see the evaluation plan implemented. 

The basic lesson learned has to do with the good faith 
effort of public school personnel who are committed to insuring 
that the RHSE will be monitored on a regular and sustained basis. 
The individuals involved took part in in-service training 
sessions and read appropriate research and evaluation literature 
to provide a context for understanding their own situation. 
Furthermore, they designed a pilot study and took extensive 
records in order to capture exactly what took place in the summer 
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program. As the evaluator, I prepared a report describing the 
evaluation plan for the RHSE and the summer pilot program. 

After one has completed an evaluation report, the basic 
question facing the evaluator is "in what ways might this report 
be useful?" One way to conceptualize the final report is as an 
educational tool regarding various modes of data, written text, 
scores, document analysis and the like for other evaluators to 
learn from. The opinion of many students, teachers, 
administrators and consultants is that the qualitative data 
collection techniques capture exactly what is going on as 
understood by participants in the program. It captures the 
complexity and wholeness of the program. It captures the 
tensions and the surprises. In effect, it provides a cultural 
description and analysis for all participants to read, reflect 
upon and, learn from. The reporting of a qualitative evaluation 
from the fourth generation perspective is socio-political in 
nature. It is meant to be diagnostic, change-oriented if 
appropriate, and educative for all parties concerned. It is 
responsive, in that all claims questions, concerns, and issues 
come from the stakeholding audiences. The evaluator is a 
negotiator in this arena. Success is assessed in terms of the 
stakeholders 1 understanding of the program and its effectiveness. 
The quality of information depends on the various participants 
themselves. The qualitative methods of evaluation used in the 
Pilot study allowed us to understand program effectiveness in a 
more realistic, authentic, and educative manner. 
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This evaluation fallowed certain rules of thumb in hind 
sight, which may be helpful to those who will conduct evaluations 
of this ty^e. The three major lessons learned included: 

(1) Know your audience, 

(2) Stay close to the data, and 

(3) don't include everything: Select the material which 
serves to answer your two major questions: 

(a) What did we do? 

(b) How well did we do this? 

As a negotiator, the evaluator becomes part of the decision 
making process. This involves time, effort and human commitment 
which can best be evaluated in an atmosphere of openness, 
understanding, authenticity; and discovery. By keeping in 
contact with the clients despite the budget cuts, the evaluator 
remains part of the project anticipating future implementation of 
the evaluation plan. 
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PROTOCOL A: 
PROTOCOL B: 

PROTOCOL Ci 



Life History 

Teacher Evaluation Survey of In-Service Training 
(Rimm, 1982) 
Parents Survey 
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PROTOCOL A: My Life History in Teaching 
Please write your own history as a teacher* Some guideline 
questions you might consider in your writing include: 

1. What do you enjoy about teaching? 

2. What do you not enjoy about teaching? 

3. When did you first decide to become a teacher? 

4. What effect do you see you have on students? 

5. What would you change, if anything, in your current role as 
a teacher? 

6. Why are you a teacher? 

7. Can you describe your teaching style? 

8. Which students do you enjoy teaching and why? 

9. What do you bring to teaching? 

10. What do you see yourself doing in the next 5-10 years? 
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PROTOCOL B: Teacher Evaluation Survey 
of In-Service Training 
(Rimra, 1982) 

EVALUATION FORM: 

Circle the appropriate number below: 
1. I found this program to be... 



1 

Dull 



of Average 
Interest 



5 

Very 
Interesting 



2. I think what I heard will be... 



Useless 



Somewhat 
Useful 



5 

Very 
Usef\ 



3. I would like to have more in-service programs on this topic. 



No, not Yes, but not for 
at all a while 



Yes, more 
soon 



4 

Other 
(Explain below) 



Explanation^ 



4. The things I liked most about this in-service were: 



5. The things I liked least about this in-service were: 
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PROTOCOL C: Parents Survey 

Parents * Reactions 

Circle the appropriate numbers below: 

1. Based on my child's experience in the RHSE, I would say the 
program was: 

1 2 3 4 5 

Not very Somewhat Extremely 

Successful Successful Successful 



2. I would like my child to continue in the RHSE, should it be 
continued. 

12 3 
Yes No Other (Explain) 



Explanation: 



3. The things I like most about my child 1 s experience were: 



4. The things I liked least about my child f s experience were: 
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